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Executive Summary 

Digital tools and technology-based activities offer new and promising opportunities for students to 
actively explore mathematical concepts and ideas in ways supported by current reforms and visions of 
mathematics instruction. This report provides an in-depth look at the implementation of SunBay Digital 
Mathematics (SunBay Math) during the second year of an i3 validation project, in two large Florida 
districts. SunBay Math is a set of middle-school curriculum replacement units centered on the use of 
technology-based, dynamically linked representations to learn core mathematical concepts. We focus 
specifically on patterns and relationships between instructional practices and instructional quality in 26 
videotaped lesson enactments that were purposefully collected to represent variation in 
implementation. 


Chapter 1 describes the conceptual framework and context for the study. As part of the scale-up and 
evaluation of SunBay Math, two units were implemented in sixth-, seventh- and eighth-grade 
classrooms in 30 middle schools randomly selected from two large Florida school districts serving 
diverse, high-need student populations. 


To study the implementation of SunBay Math lessons in this context, we draw on the constructs of 
textbook integrity (Chval, Chavez, Reys, & Tarr, 2009) and fidelity to the author’s intended lesson (Brown, 
Pitvorec, Ditto, & Kelso, 2009) to consider two dimensions in the enactment of SunBay Math lessons in 
relation to the curricular design: structural integrity and pedagogical integrity. We define structural 
integrity as adherence to the curriculum in terms of the observable parts of a lesson and activity 
structures, while pedagogical integrity refers to the interactions and conditions put in place for students 
to engage with the content of the lesson. SunBay Math’s dynamic technology-based tools feature 
multiple, linked mathematical representations designed to support student understanding of both 
underlying concepts and procedures. The design of the lessons in the modular two-week units revolves 
around teachers facilitating student reasoning through small group exploration with dynamically linked 
representations and whole group discussions to elicit, highlight and consolidate the important 
mathematical ideas. 


In Chapter 2, we present the research methods and findings related to the structural and pedagogical 
integrity of the 26 videotaped lesson enactments. We developed a set of four rubrics to focus on specific 
features of SunBay Math and also used four rubrics from the validated Instructional Quality Assessment 
(IQA) toolkit (Boston, 2012) to focus on overall instructional quality. Applying these eight rubrics to 
every lesson allowed us to capture variation in different dimensions of both structural and pedagogical 
fidelity, along with students’ overall opportunity to engage in high-level reasoning around the 
mathematics. The analysis shows that while SunBay Math lessons in the written curriculum materials 
reflect a high level of cognitive demand, many of the observed lessons were enacted at a low level, 
focusing primarily on procedures rather than reasoning and conceptual understanding. Patterns across 
both high- and low-level implementations highlight how pedagogical integrity to the lesson influences 
the opportunity for students to engage in reasoning and make conceptual connections to the 
mathematics. Through both quantitative and qualitative analysis, we demonstrate that pedagogical 
integrity is more consequential than structural integrity for students’ overall opportunity to learn. 


To further explore and contextualize pedagogical integrity and the role of teacher instructional 
practices, four case studies of lesson enactments from the same unit are presented in Chapter 3. While 
these lesson enactments focus on the same mathematical content, they reflect differences in activity 


structures (e.g., whole group vs. small group) and both high and low pedagogical integrity to the 
curriculum design. We look across the cases to highlight the role of the teacher’s approach to 
technology use, implementation of Predict-Check-Explain (PCE) inquiry cycles, and discourse practices. 
In addition, the cases illustrate important differences in teacher orientations towards their own role and 
towards the use of the curriculum. 


Chapter 4 discusses implications of the findings. We conclude that giving students access to new 
technological tools and having them work in groups to complete tasks as sequenced in the curriculum, is 
not sufficient to ensure opportunity to learn in the ways intended by the curriculum designers. The way 
that the teacher presented, facilitated and orchestrated discussion around the tasks, resulted in the 
technology being used procedurally, for exploration without any guidance, or for exploration along with 
reasoning, explanation and connections to important mathematical ideas. Whether or not teachers 
made structural or pedagogical adaptations was less important than the capacity for teachers to be able 
to do so with integrity to the curricular design. Teachers play a central role in enactment of technology- 
based curricula. We therefore underscore the need for sustained efforts to build teacher capacity, 
including opportunities for teachers to develop and refine new instructional practices, as well as develop 
new orientations to teaching and learning. 


Chapter 1 Context for the Study 


Recent visions of mathematics instruction focus on engagement in productive struggle as students make 
sense of and reason about mathematical ideas to develop conceptual and procedural understanding 
(Common Core State Standards Initiative, 2010; Hiebert et al., 1996). These new views of mathematics 
instruction are often regarded as ambitious, highlighting the challenges involved in taking on new roles, 
responsibilities, and practices for both teachers and students (Franke, Kazemi, & Battey, 2007; Lampert, 
Beasley, Ghousseini, Kazemi, & Franke, 2010). Innovative curricula are viewed as an essential lever for 
reform in these directions, providing support for new standards and instructional methods (Remillard, 
2005). 


The growth of new digital tools and technology-based activities provides new opportunities for students 
to explore mathematical concepts and ideas through dynamic visual representations and simulations 
not possible with traditional paper and pencil exercises or chalkboard demonstrations. Technology 
enables transformation in learning experiences by supporting students in reasoning about mathematical 
concepts through experiences in intrinsically engaging environments (Roschelle, Noss, Blikstein, & 
Jackiw, 2017). Dynamically linked mathematical representations afford opportunities for the 
development of deep conceptual understanding of mathematical ideas promoted by current reforms. In 
this report, we explore the implementation of SunBay Digital Mathematics (SunBay Math), a set of 
middle-school curriculum replacement units centered on teacher facilitation of the use of technology- 
based, dynamically linked representations, to help students learn core mathematical concepts. 


These replacement units reflect a shift in the teacher’s role to engage students in inquiry and facilitate 
discussions to develop a shared understanding of important mathematical ideas. We explored what 
happens during implementation when teachers use curriculum units that reflect new pedagogical 
approaches and incorporate technology. Questions guiding our exploration included: Does providing 
teachers and students access to technology-based activities lead to increased opportunities for deep 
engagement and opportunities to learn? If so, under what conditions? Do teachers’ instructional 
practices reflect the intent of the curriculum and technology? 


This study is part of a larger external evaluation of the impact of SunBay Math in two large districts in 
Florida, which included a mixed methods implementation study and a randomized control trial.* Findings 
are described in a companion report, The i3 Validation of SunBay Digital Mathematics (Sirinides et al., 
forthcoming). The in-depth analysis from the evaluation showed a positive impact on the teachers’ 
understanding and tolerance of student struggle as a part of the math learning process, as well as an 
increase in the teachers’ comfort with using technology for math instruction. However, many teachers 
did not fully implement the SunBay Math units and/or lessons, making it difficult to assess the true 
impact of the innovative technology-based curriculum on student learning. 


1 In 2014, SunBay Math’s developer, SRI International, was awarded an i3 validation grant as part of the Investing in Innovation 
(i3) program of the Office of Innovation and Improvement (Oll), U.S. Department of Education. The Helios Education 
Foundation also supported the project with a matching grant. As part of this award, the Consortium for Policy Research in 
Education (CPRE) at the University of Pennsylvania, served as an independent evaluator, assessing the impacts and 
implementation of SunBay Math under the i3 project in two districts. 


The present study draws on prior work that has shown that teacher practices can mediate the effects of 
curriculum on student learning (Tarr et al., 2008; Taylor, 2016). It examines dimensions of variation in 
SunBay Math lesson enactments and teacher instructional practices that are associated with that 
variation. It also looks at pedagogical and structural factors associated with students’ opportunities to 
engage in reasoning and making conceptual connections to the mathematics. This report aims to 
contextualize and further explain the effects of teacher adaptations on the implementation of 
technology-based middle-school mathematics curriculum. 


Conceptual Framework 

SunBay Math is comprised of professional development and instructional resources that enable 
teachers to engage students in sense-making and reasoning—important priorities in current efforts 
towards reform in mathematics education. Because “instruction consists of interactions among teacher 
and students around content, in environments” (Cohen, Raudenbush, & Ball, 2003, p. 122), the effects 
of instructional resources are always mediated by teacher and student knowledge, beliefs and action. 
Teachers and students adapt curriculum materials during implementation and these adaptations may 
enhance or detract from curriculum developers’ intent (Henningsen & Stein, 1997; Otten & Soria, 2014; 
Remillard, 2005; Stein, Grover, and Henningsen, 1996; Stigler & Hiebert, 2004). Furthermore, a high 
level of cognitive demand is difficult to maintain and often declines during enactment, particularly with 
curricula focused on reasoning and problem-solving, where enactment is more dependent upon the 
interactions between teachers and students (Stein, Remillard, & Smith, 2007). 


Implementation of curricular interventions is often studied in terms of fidelity and measured by 
observable features such as the extent of use, coverage, or faithfulness to the text (Brown, Pitvorec, 
Ditto, & Kelso, 2009). Yet research demonstrates that curricula may be used in ways that incorporate 
superficial features but do not reflect the underlying pedagogical principles of the curriculum design 
(Coburn, 2003). Brown (2009) conceptualizes teachers not as adopters or users, but as designers, 
emphasizing the “dynamic interplay” between the teacher, technology and written curriculum 
materials. Teachers interact with the curriculum to select, interpret, and reconcile the intended goals 
with their own goals and the perceived constraints, as well as to accommodate instruction to their 
students. In addition, teachers use curriculum in different ways: they may follow it as a script; they may 
work from the materials but adapt; or they may improvise by relying more on their own ideas and 
resources than what is in the written curriculum (Remillard & Bryans, 2004). Important to this process is 
the teacher’s pedagogical design capacity, or "skill in perceiving the affordances of the materials and 
making decisions about how to use them to craft instructional episodes that achieve her goals" (Brown, 
2009, p. 29). This view of the teacher’s relationship with the curriculum places the teacher in an active 
role of participation with the curriculum, where there is a bi-directional or dynamic interchange 
between the teacher and the text (Remillard, 2005). 


In conceptualizing the implementation of SunBay Math in classrooms, we view teachers as participating 
with the curriculum (Remillard, 2005) and draw upon recent work that problematizes the practice of 
evaluating implementation of curricular interventions only in terms of fidelity. Chval, Chavez, Reys, and 
Tarr (2009) propose an alternative construct called textbook integrity that includes not only regular use 
of the curricular materials and how much of it is used but also the consistency of the instructional 
strategies with the pedagogical orientation of the materials as written. Similarly, at the lesson level, 
Brown et al. (2009) propose that there are two different forms of fidelity to consider: fidelity to the 


literal lesson or the degree to which the lesson is implemented as written and fidelity to the authors’ 
intended lesson, defined as “the degree of alignment between the author’s intended opportunities to 
learn and the opportunities to learn in the enacted lesson” (p. 373). In defining the authors’ intended 
opportunity to learn, they draw upon Hiebert’s (2003) notion of considering not just exposure to the 
content, but also the nature of students’ engagement with content in particular ways (e.g., reasoning, 
communicating, exploring). 


In our study, we focus specifically on lesson enactment and draw upon Chval et al.’s (2009) notion of 
integrity along with Brown et al.’s (2009) conceptualization of fidelity to the author’s intended lesson, to 
consider two dimensions in the enactment of SunBay Math lessons: structural integrity and pedagogical 
integrity. Structural integrity is adherence to the curriculum in terms of the observable parts of a lesson 
and activity structures (e.g., the Warm-Up vs. the Main part of the lesson, whole class vs. small group 
learning). Pedagogical integrity refers to the interactions and conditions put in place for students to 
engage with the content of the lesson in ways envisioned by the curriculum designers (e.g., reasoning 
about mathematical ideas through exploration with dynamically linked representations). One way to 
think about these distinctions is that the structural integrity of a lesson could be determined by 
watching a lesson but without hearing any teacher or student voices; whereas pedagogical integrity 
requires listening to what teachers and students are saying and how they are interacting and 
responding, both to each other and to the content of the lesson. By focusing on these constructs of 
structural and pedagogical integrity in this empirical study, we aim to identify the conditions and 
features of lesson enactment that help students make conceptual connections to the mathematics. This 
analysis may help improve the implementation of SunBay Math and similar technology-based curricular 
interventions. 


Curriculum Design 

SunBay Math was designed as a curricular activity system in which technology-based dynamic 
mathematical representations, standards-based instruction, and teacher professional development are 
integrated to meet the needs of local educational contexts (Vahey, Knudsen, Rafanan, & Lara-Meloy, 
2013). The program consists of two- to three-week long curriculum replacement units designed to 
improve students’ understanding of key math concepts (functions, ratio and proportion, algebraic 
expressions and geometric transformations) and incorporate effective instructional practices and 
mathematical practices from the Common Core State Standards for Math (CCSSM). By introducing 
teachers and students to new ways of learning and interacting with complex mathematical concepts and 
practices through a modular design, the units were designed as replacement units so that they could be 
easily implemented and also help facilitate instructional reform (Davis & Krajcik, 2005; Ball & Cohen, 
1996). The theory behind the use of replacement units is that once teachers begin to see the resulting 
depth of student learning and engagement, they will be motivated to make broader changes in their 
math instruction, and perhaps even use their regular curriculum materials differently (e.g., focus on 
conceptual understanding, provide opportunities for collaborative learning, support productive struggle, 
and so forth). 


SunBay Math’s dynamic technology-based tools feature multiple, linked mathematical representations 
that are designed to support student understanding of both underlying concepts and procedures. Each 
unit is designed around the use of one of four dynamic web-based tools for generating and manipulating 
mathematical relationships through simulations. For example, Ratio Visualizer—explored in more detail 


in the case studies in Chapter 4—includes dynamic visual and graphical representations of ratio and 
proportions through a simulation of paint mixing with pips or units of color to make various shades of 
paint. There are also tools for dynamic, linked representations of algebraic expressions, geometric 
transformations and linear functions. 


Shown below in Figure 1 are four dynamically linked representations in the Ratio Visualizer tool used in 
the sixth-grade unit on ratio and proportion. The mixer (shown in the top images) allows the user to 
enter any number of paint pips (dots) in each color. The resulting color blend is shown by a triangular 
marker on the spectrum at the bottom of the screen. Directly below the spectrum, the blend bar shows 
the number of pips of each color in a ratio bar that corresponds to the spectrum. In the first image, the 
ratios 10:14 and 5:7 are represented on both the blend bars and the spectrum. If the ratios were not 
equivalent, the blend bars would not line up. Users can also link to and manipulate the container 
representation to organize the pips into rows and columns. In the image on the top right, the container 
has been resized to show 10:14 as two equal rows of 5:7, which is also shown on the spectrum and 
blend bars. The container therefore allows the user to see the unit ratio, or the ratio in its simplest form, 
within any ratio. Multiple ratios and equivalences can also be represented in a graph and table format as 
shown in the image at the bottom left. Finally, at any time the user can select Show Artwork to pull up a 
picture of the artwork along with images of the color blends that have been entered (bottom right). 


Figure 1. Ratio Visualizer Dynamic Representations: Linked Mixer, Spectrum, Container, Graph and 
Artwork 
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Users can manipulate specific inputs (e.g., adding units of paint to create a new color, or in geometry, 
stretching a shape to alter its angles) to produce linked changes across the multiple visual 
representations. This linked structure allows students to explore underlying mathematical relationships 
by observing how different inputs produce different responses across the various representations. These 
tools therefore allow for representational fluency, or the ability to make meaning of and move between 
different representations of a mathematical concept (Zbiek, Heid, Blume, & Dick, 2007). 


The SunBay Math units consist of 7 to 10 investigations, most of which are designed to be completed in 
one 45-minute class period. Each investigation contains four parts: (1) a short Warm-Up section 
(problems designed to preview the math needed for the lesson), (2) the Main section (a set of tasks 
designed to be completed through inquiry, using the technology), (3) the Wrap-Up (teacher facilitated 
discussion to make the math explicit and consolidate learning) and (4) Problem-Solving (problems that 
can be started in class and then assigned for homework). Each unit has a student workbook in which 
students can read the tasks and record their responses while working with the technology. The SunBay 
Math Teacher Guides include sections on the mathematical goals, standards, key ideas, implementation, 
as well as suggested timings, tips, teacher questions and student answers for each investigation. 


SunBay Math is designed to support students in an inquiry cycle of predicting, evaluating and 
understanding mathematical change through exploration with dynamic, linked representations. The 
Predict-Check-Explain (PCE) cycle is designed to allow students to develop and justify conjectures or 
hypotheses about the impacts of various inputs across different representations, to test the hypotheses 
using the technology, and then to reflect on and discuss the accuracy of their hypotheses and the 
insights they developed through the cycle. As one member of the development team explained: 


Predict requires you to bring your prior knowledge—you have to start with whatever 
understanding you have and then you check with the software. It's not the teacher 
who's telling you that you're correct or not correct, it's the software that actually 
represents it. Then you have to explain why things are happening in the way that they 
are, from the software. 


The designers of SunBay Math present this approach as a contrast to purely symbolic approaches to 
traditional mathematics instruction, where mathematical rules are learned in isolation and then applied 
to various situations. Classroom teachers facilitate this process by guiding students’ progression through 
multiple PCE cycles during each lesson. 


In addition to the structural components of a SunBay Math lesson (i.e., using the technology and 
workbooks and completing the lesson sequence), SunBay Math expects teachers to engage learners ina 
manner that positions them to think through problems themselves, as opposed to doing most of the 
intellectual work for students. The teacher is positioned in the role of a facilitator, leveraging the 
perspectives and ideas of individual students while guiding the entire class toward deep conceptual 
understanding of mathematical concepts. As one of the curriculum developers stated, the teacher’s job 
is to “get the students to learn it in a way that makes sense, not to memorize it.” 


This intentional framing of teachers as facilitators pushes against common practices found in many math 
classrooms where teachers tell or show students procedures and then have students complete practice 
problems, often without students acquiring any deep conceptual understanding of what they are 
learning (Stigler & Hiebert, 1997). The teacher’s role in SunBay Math involves actively leading whole 


class introductions to build on students’ prior knowledge and prepare them to complete the tasks 
successfully through the PCE cycle. Students then engage in PCE cycles with the technology in small 
groups, as teachers circulate through the classroom. They listen in on students’ dialogues about their 
experimentations with the technology, and interject with strategic questions, probing students’ thinking 
to lead them toward mathematical reasoning, while simultaneously ensuring that they move through 
the investigations in a timely manner. At the end of the lesson, teachers again orchestrate whole class 
discussions to elicit student strategies and reasoning around the important ideas, steering student 
contributions toward a shared understanding of the central mathematical concepts. Thus, the teacher’s 
role involves some restraint in not directly telling students what they need to see and learn, but also 
careful listening to build on and support students’ mathematical reasoning through questioning and 
probing to ensure that they learn important mathematics. A curriculum developer explained that ina 
SunBay Math lesson you should see “students talking more than teachers,” while teachers are asking 
lots of open-ended questions with a “repeated emphasis on ‘why’.” Consequently, implementing the 
curriculum in line with the developer’s intent is a matter of orchestrating specific structural components 
and activity structures as well as relatively sophisticated pedagogical moves. 


Implementation 

As part of the scale-up and evaluation of SunBay Math, two units each for Grades 6, 7 and 8 were 
implemented in 30 middle schools randomly selected from two large school districts (District 1 and 
District 2) with diverse, high-need student populations in Florida. During 2016-2017, the second year of 
implementation when the evaluation took place, the District 1 student population comprised 39% 
African-American and 32% Hispanic students, with 61% eligible for free or reduced-price lunch. The 
District 2 student population comprised 32% Hispanic and 28% African-American students, with 59% of 
students eligible for free or reduced-price lunch. 


The developers of SunBay Math created the online tools and supporting materials as well as “train the 
trainer” modules for district-level facilitators who had considerable experience as math teachers and/or 
school-based math coaches. This training focused on the content of each unit, while also being designed 
to give the facilitators a sense of the experience students should have when engaging with the 
technology and the tasks. The facilitators then provided 2.5 days of professional development sessions 
for teachers of Grades 6-8 math (1.5 days prior to using their first SunBay Math unit, and an additional 
half-day prior to implementing any new unit for the first time), which reflected not only the content and 
experiential aspects, but also some additional focus on pedagogical strategies. Teachers were expected 
to implement two 2-week SunBay Math modules during the school year in place of the regular curricular 
materials, over the two-week periods when the corresponding concepts or standards were covered in 
the district pacing guidelines. 


During the school year, teachers also received on-site support from the facilitators who were available 
to provide coaching to teachers via conferencing, observation, reflection, modeling, or any other type of 
support that teachers requested. All but two schools also had access to support from one or more 
school-based teacher leaders who regularly met with implementing teachers to reflect on and discuss 
SunBay Math implementation. This model of implementation therefore included several layers of 
support and potential points of translation from the curriculum developer’s vision to the actual 
enactment in classrooms. 
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Chapter 2: Methods 


Data Collection and Participants 

Videorecorded lesson enactments were collected from a purposive sample of 26 Grade 6-8 classrooms 
in schools from both Florida districts implementing SunBay Math during Year 2 of the evaluation. 
Drawing on the expertise of the district facilitators, the sample was designed intentionally to elicit a 
range of lesson enactments in relation to district, grade level, unit, teacher experience and the 
facilitator’s view of the overall strength of the implementing teacher. Participation in video recording 
was voluntary and as Tables 1 and 2 show, the resulting sample reflects variation in all but one 
dimension: there were only 4 teachers from District 2. Only one lesson from each teacher was recorded; 
the sample was not selected to be representative of the larger implementation nor representative of the 
particular teacher’s approach to instruction, but rather to capture and explore variation in the 
enactment of SunBay Math lessons in classrooms. 


Table 1. Characteristics of Teachers who were Videorecorded (n = 25/26*) 


Role District Teaching SunBay Math Highest Degree 
Experience Experience Received 
Teacher 22 District1l 22 1-5 years 6 1 year 10 Bachelors 16 
Teacher 3 District2 4 6-11 years 5 2 years 15 Masters 9 
Leader 12+ years 14 


*We did not have information on one teacher's background. 


Table 2. Videorecorded Lesson Enactments 


SunBay Math Topics and Units Grade Videorecorded Number of 
Lessons Investigations 

Ratio & Proportion 

True Colors Murals 6 10 6 

Managing the Soccer Team 7 2 1 

Algebraic Expressions 

Little X Games 6 2 2 

3D Design Studio 7 7 3 

Transformational Geometry 

Transformation Nation 8 5 3 


The 26 videorecorded lessons reflect 15 different SunBay Math investigations from five of the six SunBay 
Math units. Table 2 shows the variation in the sample of 26 video recordings collected across grade 
levels and units. While the sixth- and seventh-grade units were in the second year of implementation, 
the eighth-grade units were being implemented for the first time. 
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Data Analysis 

Analysis of the lesson enactment data proceeded in three stages. In the first stage, our goal was to 
characterize the dimensions on which lesson enactments varied in relation to both components unique 
to SunBay Math and to overall instructional quality. Four rubrics were selected from the Instructional 
Quality Assessment (IQA) (Boston, 2012; Matsumura, Slater, Junker, & Peterson, 2006), a validated 
toolkit that is designed to measure the quality of instruction in relation to cognitive demand (see 
Appendix A). The IQA Potential of the Task rubric is designed to measure the level of cognitive demand 
in the lessons as per the teachers’ guide. Implementation of the Task holistically assesses the level of 
thinking students actually engaged in throughout the lesson, while Rigor of Discussion and Rigor of 
Teacher Questions relate specifically to the whole class discussion that occurs after students have had 
the opportunity to engage in mathematical work. While only a portion of any given SunBay Math lesson 
is designed to be whole group, it is during these discussions that students have opportunities to engage 
in reasoning and explanation, consolidate learning, and draw connections between different strategies. 
As the developer of the IQA (2012) asserts, “The whole-group discussion provides an opportunity for 
teachers to advance the mathematical understandings of a// students” (Boston, 2012, p. 83). 


We also viewed a subset of videos to determine dimensions that varied in lesson enactment in relation 
to central characteristics of SunBay Math as it is designed (i.e., elements that might not be as important 
in the enactment of other curricular materials). Four categories emerged from this analysis: 1) the 
presence of the three core components of a SunBay Math investigation (Warm-Up, Main, and Wrap-Up), 
2) the enactment of the PCE cycle, 3) the approach to technology use and 4) the activity structure of the 
lesson in terms of balance between teacher facilitation and opportunities for student collaboration. 
Through an iterative process, we constructed four SunBay Math Enactment rubrics to capture levels of 
variation along these dimensions as shown in Appendix B, piloted them with a subsample of videos, and 
then further refined them. 


In the second stage of analysis, a team of three researchers applied the four SunBay Math Enactment 
rubrics and four IQA rubrics to the entire sample of 26 videorecorded lessons. The team first watched, 
coded and discussed 12 videos as a group to establish consistency in coding. Each video was then coded 
by at least two members of the research team. On a weekly basis, all three video coders met to discuss 
and resolve any discrepancies in the coding. 


In the third stage of analysis, we aligned the rubric levels with the key indicators of the curriculum 
design, or the curriculum designer’s intent (Brown et al., 2009), in order to operationalize our constructs 
of opportunity to learn, structural integrity and pedagogical integrity. 


Opportunity to learn was captured by the IQA rubric for Implementation of the Task, which assesses the 
overall opportunity students have to engage in high-level thinking and reasoning. Structural integrity 
was revealed in the extent to which the lesson began with a Warm-Up and a whole group discussion of 
the task, followed by students working in small groups on additional tasks with the technology while the 
teacher circulated, concluding with a whole class discussion or Wrap-Up. Pedagogical integrity to the 
curriculum designers’ intent was indicated in the use of technology for exploration and the 
incorporation of the PCE cycle throughout the lesson, along with the scores on the IQA for teacher 
questions and student discussion. These last two IQA rubrics hone in on teacher and student actions in 
shaping mathematical discourse during whole group discussions. 
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These seven rubrics were then used to define high, moderate and low levels of opportunity to learn, 
structural integrity and pedagogical integrity, in relation to the curriculum design, as shown in Table 3, 
and described in more detail below. 


Table 3. Dimensions of Opportunity to Learn, Structural Integrity and Pedagogical Integrity to the 
Curriculum Designer’s Intent 


Opportunity to Learn Structural Integrity Pedagogical Integrity 


Level of IQA Rubric Levels for Balance Three-part Approach PCE IQA Rubric Levels for 
Integrity Implementation of Activity Investigation to Questioning and 
Structures Structure Technology Student Discussion 
Use 
High Doing mathematics or Mostly All three Exploration Consistent Doing mathematics 
procedures with small connected with cycles or procedures with 
connections group components connections connections 
Moderate Procedures without Mostly Atleast two Exploration Atleast one Procedures without 
connections whole connected without cycle present connections 
group components guidance 
Low Memorization Entirely Only one Procedural Partial cycle Memorization 
small or component use 
whole 
group 
None Absence NA No Not used None Absence 


components 


Determining Levels of Opportunity to Learn 

The IQA Implementation of the Task rubric (see Appendix A) focuses on the overall level of cognitive 
demand that students are engaging in during the lesson. At the highest score level of 4 (doing 
mathematics), students are “exploring and understanding the nature of mathematical concepts, 
procedures, and/or relationships” and there is explicit evidence of student reasoning and justification. 
At level 3 (procedures with connections), students are “creating meaning for mathematical concepts, 
procedures, and/or relationships” but there is no explicit evidence of student reasoning or justification. 
Levels 3 and 4 were considered to be high in relation to the curriculum designers’ intent, as both of 
these levels reflect an overall focus on students’ reasoning and making conceptual connections. Level 2, 
procedures without connections, was considered a moderate opportunity to learn, where students are 
using procedures that are “explicitly called for’ and focusing on producing correct answers rather than 
connections. Finally, level 1, where students are recalling or reproducing memorized facts, rules, 
formulae, or definitions, was considered to be a low opportunity to learn. 


Determining Levels of Structural Integrity 
The two dimensions of structural integrity were assessed by applying the rubrics for: 1) balance of 
activity structures and 2) investigation components. The SunBay Math lessons are written to reflect a 
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combination of whole class discussion, group work and individual work. Enactments that had some 
teacher-facilitated whole group portions with extended opportunities for students to work in groups, 
most closely reflected the balance of activity structures as written in the curriculum, and were 
considered high integrity. If the enacted lesson was mostly whole group but had at least some 
opportunities for student collaboration, it was considered to be at a moderate level of integrity, while 
those that were entirely whole group or entirely small group were considered to be of low integrity to 
the curriculum designer’s intent. Additionally, lesson enactments that had the three main components 
from the same investigation (i.e., Warm-Up, Main, and Wrap-Up) were considered high integrity, while 
those that had two or one were considered medium and low respectively. 


Determining Levels of Pedagogical Integrity 
The four dimensions of pedagogical integrity were assessed by application of four of the rubrics: 1) 
approach to technology use, 2) PCE cycles, 3) teacher questions and 4) rigor of discussion. 


For approach to technology use, our category of Exploration with Connections was considered the 
highest level of integrity to the authors’ intent. This category includes teachers who intentionally guided 
and focused students on using the dynamic representations to reason, explain and construct arguments 
about the mathematics without following a specific procedure or solution path. Moderate integrity was 
characterized by the category Exploration without Guidance where teachers provided opportunities for 
students to use the technology to solve tasks and potentially to make connections with the dynamic 
representations, but there was no explicit evidence of teacher guidance or evidence that students were 
engaging at that level. Procedural Use of Technology was considered low integrity, where the dynamic 
representations in the technology were used by teachers and students primarily to confirm a solution 
already determined by a procedure or where students were directed by the teacher to a specific 
solution path or method. 


For PCE, completion of all or most PCE cycles in the lesson enactment represented the highest level of 
integrity to the curriculum designer’s intent, since this was a central feature of the design of the 
investigations and tasks. Moderate integrity was characterized by the presence of at least one PCE cycle 
during the lesson, while incomplete cycles (e.g., Predict and Check without Explanation) were 
considered low integrity. 


The third and fourth dimensions of pedagogical integrity were determined by converting the IQA scores 
on the rubrics for Teacher Questions and Rigor of Discussion (see Appendix A) into levels of low, 
moderate and high pedagogical integrity. 


The two IQA rubrics consist of a scale from 0 to 4, representing a continuum of increasing cognitive 
demand through opportunities for students to engage in reasoning, connections and explanations. Table 
4 illustrates how we used the descriptors of these rubric scores to convert them into low, moderate and 
high levels of integrity to the curriculum design. As the authors of the IQA state, “a demarcation line 
exists between score levels 2 and 3 that separates high- and low-level cognitive demands, talk moves, or 
expectations in each dimension” (Boston 2012, pp. 85-86). Therefore, ratings on these rubrics had to be 
at least at level 3 to be considered high pedagogical integrity. 


The Rigor of Discussion rubric measures the level of reasoning, explanation and justification that 
students engage in during whole group portions of the lesson. This is influenced by the questions that 
teachers pose to elicit student thinking. However, even when teachers pose high-level questions, 
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students do not always respond with complete or substantive explanations. Together, these two rubrics 
provide a snapshot of the discourse in terms of the level of discussion occurring in a lesson. For these 
dimensions of discourse, we considered the highest level of Doing Mathematics to be going above and 
beyond the intent of the curriculum as designed, and designated that score as High+ in terms of 
pedagogical integrity. 


Table 4. IQA Score Levels in Relation to Integrity to the Curriculum Design 


Integrity Score Level of Cognitive Characteristics of Discourse 
Demand 
High+ 4 Doing mathematics Explicit explanations, connections or reasoning; high quality 


talk moves and student responses 


High 3 Procedure with Substantive attempts to engage students in explanations, 
connections to meaning connections or reasoning, but these may be incomplete 

Medium 2 Procedures without Focus on rote or procedural skills; weak or minimal attempts 
connections to meaning at talk moves 

Low 1 Memorization Focus on facts, rules, or formulas; absence of talk moves or 


more than one-word responses from students 


NA 0 Absence Absence of whole group discourse 


To determine if there were relationships between students’ engagement and opportunity to learn in the 
enacted lesson and the dimensions of pedagogical and/or structural integrity, we analyzed the levels of 
integrity of each dimension (shown in Table 3) across the individual lesson enactments. We then 
purposefully selected four lesson enactments from the same or similar investigations in one unit to 
serve as case studies of lesson enactment (Patton, 1990). The rationale for the selection of the four 
lesson enactments and additional methods of case study analysis are described in Chapter 4, the case 
study section of the report. 
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Chapter 3: Structural and Pedagogical Integrity of Enacted Lessons 

In this study, we examined the enactment of SunBay Math lessons in terms of: (1) opportunities for 
student engagement with mathematical concepts and; (2) teacher practices associated with enacting 
these lessons according to the curriculum designers’ intent. When we compared the IQA scores for 
Potential of the Task with Implementation of the Task (see Appendix C), we found that in line with many 
other studies of lesson enactment, most of the lessons (20 out of 26) were enacted at a lower level of 
cognitive demand than the level at which they were written in the curriculum materials (Cohen, 1990; 
Henningsen & Stein, 1997; Lloyd, 1999; Stigler & Hiebert, 2004). The preponderance of video-recorded 
lessons were enactments at a level 2 on the Implementation of the Task IQA rubric, or Procedures 
without Connections (n = 16), indicating that proceduralizing high-level tasks was common. 


Based on our sample of classroom observation data, we also found that teacher adaptations were 
common, both in the ways the lesson was structured for students and in the instructional moves that 
shaped students’ interaction with the content of the lesson. Table 5 presents the frequency of levels of 
pedagogical and structural integrity along with the overall level of implementation across the 26 
observations. 


Table 5. Levels of Pedagogical Integrity, Structural Integrity, and Opportunities to Learn 


Absent Low Med High 


NA* 0 (1) (2) (3/4) 

Pedagogical Integrity 

Approach to technology use 2 12 5 

Predict, check, explain 1 10 5 

Teacher questions (IQA) 0 12 6 

Student discussion (IQA) 4 9 4 
Structural Integrity 

Balance of activity structures 0 6 14 6 

Investigation components 0 3 13 10 


Opportunity to Learn 

Implementation of the Task (IQA) 0 0 16 10 
Notes: The eighth-grade transformational geometry unit had some introductory 
investigations that did not explicitly require the use of technology or PCE; therefore 
these lessons were given NA for those rubric scores. 


As Table 5 illustrates, the lesson enactments varied across all dimensions of pedagogical and structural 
integrity as well as opportunity to learn. Most of the lessons were implemented at a procedural level 
(16), while 10 were implemented at a high level of cognitive demand (procedures with connections or 
doing mathematics). We further explored the associations of these dimensions of structural and 
pedagogical integrity with the opportunities for students to engage in reasoning and make conceptual 
connections to the mathematics. When the levels of implementation are placed side-by-side with the 
levels of pedagogical and structural integrity, shown in Figure 2 as a heat map, some illuminating 
patterns emerge. In Figure 2 the lessons are ordered first by opportunity to learn (level of 
implementation) and then by pedagogical integrity. 
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Figure 2. Opportunity to Learn, Pedagogical Integrity and Structural Integrity Across Lesson Enactments 
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integrity. The 10 lesson enactments that were rated at a high level of opportunity to learn (in the the top 
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half of Figure 2) had either moderate or high levels of pedagogical integrity: 9 of the 10 were from either 
the sixth-grade proportions unit or the eighth-grade transformational geometry unit. In contrast, all of 
the enactments of lessons from the seventh-grade algebra and proportions units were rated at a low 
level of opportunity to learn and also had moderate to low levels across the dimensions of pedagogical 
integrity. 


This suggests that the unit itself (the mathematical content and/or the nature of the technology) may be 
an important factor in the nature of the transformations that are made during enactment, though the 
sample size is too small to be conclusive. Transformational geometry is a topic that lends itself to 
conceptual exploration while algebraic expressions is a topic that has traditionally been taught 
procedurally in middle school. In interviews conducted for the larger evaluation study, many teachers 
commented that students had a harder time understanding and making sense of the visual 
representations in the technology for the algebraic units. Alternatively, the dynamic mathematical 
representations in the proportions and geometry units may have been more conducive to conceptual 
exploration than those in the algebraic units. While it is beyond the scope of this study to make that 
determination, our findings suggest that the affordances for conceptual connections in the curriculum 
unit and/or related technological tools is an important consideration. 


To mitigate the influence of the curricular unit on implementation with integrity, we looked more closely 
at the variation within units. In the sixth-grade proportions unit, we had the most videorecorded lessons 
(10) as well as the most variation in the enacted levels of opportunity to learn, pedagogical integrity and 
structural integrity. The sixth-grade True Colors Murals unit focuses on understanding the multiplicative 
relationships within and between ratios and using these relationships to recognize equivalence, find 
missing values and compare ratios. Figure 3 shows the relationships between the investigation and the 
opportunity to learn, pedagogical integrity and structural integrity across the 10 lesson enactments from 
that unit. 


As Figure 3 shows, pedagogical integrity appears highly associated with the maintenance of high levels 
of implementation or opportunity to learn. Adaptations teachers made to the lesson structure 
(Investigation Components) and the grouping of students (Balance of Activity Structures) in that same 
unit were less influential in maintaining high levels of cognitive demand for student engagement than 
the adaptations they made to pedagogical practices. 
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Figure 3. Opportunity to Learn, Pedagogical Integrity and Structural Integrity Across Lesson Enactments 
from Grade 6 True Colors Murals Unit 
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Pedagogical practices which support the development of students’ sense-making and reasoning are 
essential to maintaining or raising the cognitive demand of the lesson, regardless of whether the lesson 
was enacted in a whole group or small group activity structure. Figure 3 shows that the enactment of 
the lessons in the first five rows was implemented at a high level and reflected the SunBay Math 
curriculum designers’ intended pedagogical emphasis on student exploration, reasoning and explanation 
around the dynamic mathematical representations in the technology. However, structural integrity to 
the lessons as written—that is, having students work in groups and following the lesson structure—did 
not, on its own, appear to provide these same opportunities to learn for students. Four of the five 
lessons rated as having a low level of implementation also have high levels of structural integrity in 
either balance of activity structures or investigation components. 


A closer examination of the five lessons enacted at lower levels of implementation (Procedures without 
Connections on the IQA) reflects an overall focus on learning procedures, getting correct answers and 
completing tasks, rather than on sense-making or reasoning. These lessons manifested lower levels of 
teacher questioning and discussion, and most were also low on PCE and the approach to technology use, 
reflecting a transformation of the SunBay Math activities and practices to align with a more traditional 
and procedural approach to mathematics instruction. In most cases, the PCE cycle was transformed into 
PC (Predict and Check) or sometimes only Check. Students had opportunities to use the technology, but 
they were using it primarily to check the veracity of their solutions rather than to explore the 
mathematics. 
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To further explore the associations between pedagogical integrity and structural integrity to the overall 
level of implementation”, Table 6 presents statistical associations between each of the rubric scores 
calculated at Kendall’s tau (Kendall & Gibbons, 1990; Kendall, 1938).? The correlation matrix presents 
coefficients for the full sample (n = 26) in the shaded triangle above the diagonal, and the same 
coefficients for the reduced sample of lesson enactments from the True Colors Mural unit (n = 10). 


Table 6 Statistical Associations among Class Observation Rubric Scores 


Rubrics LOl ATU PCE QUE DSC BAS INC 

Level of implementation 1.000 0.676 0.644 0.709 0.555 0.307 0.338 
Approach to technology use 0.796 1.000 0.721 0.676 0.630 0.412 0.338 
Predict Check Explain 0.821 0.866 1.000 0.695 0.673 0.319 0.385 
Questions 0.937 0.746 0.800 1.000 0.615 0.480 0.229 
Discussion 0.785 0.828 0.956 0.795 1.000 0.370 0.324 
Balance of activity structures 0.131 0.289 0.250 0.215 0.179 1.000 0.204 
Investigation components 0.214 0.393 0.510 0.335 0.521 0.510 1.000 


Shaded triangle above the diagonal is Kendall’s t for the full sample (n = 26). Unshaded lower triangle is for the 
True Colors Mural videos only (n = 10). 


We find that for both full and reduced samples, the level of implementation or opportunity to learn was 
significantly and positively associated with all four dimensions of pedagogical integrity: technology use, 
PCE, questioning and discussion. The coefficients ranged from 0.56 to 0.71 for the full sample, and were 
larger for the True Colors Mural unit lessons, suggesting that instances of reduced pedagogical integrity 
corresponded with decreased opportunities for students to engage in high-level thinking and reasoning. 
The dimension of pedagogical integrity with the highest estimated relationship to opportunities to learn, 
was questioning. In contrast, the correlations between the two dimensions of structural integrity and 
opportunitites to learn were not statistically significant. Thus reductions in structural integrity did not 
correspond with decreased opportunities for students to engage in high-level thinking and reasoning. 


We also looked for evidence of the relationships of these dimensions to each other. Table 6 shows a 
relatively large correlation between Approach to Technology Use and PCE. This suggests that when 
teachers use PCE as intended in the curriculum design, technology is often used in a way that supports 
student exploration of mathematical concepts. The two dimensions of structural integrity show no 
statistically significant association either with each other or with pedagogical integrity. 


Looking across these dimensions of enactment together suggests that the Approach to Technology Use, 
PCE, and discourse practices (teacher questioning and student discussion) are high-leverage pedagogical 
practices that can maintain or lower the overall level of implementation of SunBay Math in terms of 
cognitive demand, or opportunity to learn, for students. They are also highly associated and 


? Level of implementation was measured by the score on the IQA Implementation of the Task rubric (see Appendix 
A). 

3 Because of the small sample size and ordinality of the variables, a non-parametric estimator was used to test for 
independence based on ranks. Exploratory analysis was also conducted using Spearman’s rho correlation as well as 
multiple correspondence analysis. The results did not differ from those presented in magnitude or statistical 
significance. Kendall’s estimator is based on concordant and discordant pairs and usually smaller than values of 
Spearman’s rho correlation. Significance tests for tau are also more accurate with smaller sample sizes. 
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complementary practices. Dimensions of structural integrity are less influential in determining the 
nature of students’ engagement with mathematical ideas. 
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Chapter 4: Case Studies of Lesson Transformations 

We now turn to examples of four of these video enactments from the True Colors Murals unit to explore 
how the same content can be implemented to reflect different levels of both pedagogical and structural 
integrity, and to illustrate in context the particular teacher practices and orientations which are 
associated with different opportunities to learn for students. The case studies were purposefully chosen 
for contrast in both structural and pedagogical integrity to the written curriculum as shown in Table 7 
below. 


Table 7. Case Studies of Lesson Enactments from True Colors Murals 


Structural Integrity Pedagogical Integrity 
Low High 
Low Ms. Elmore 
(Inv. 4) 
Moderate Ms. Deegan Ms. Aguila 
(Inv. 2) (Inv. 2) 
High Ms. Chancellor 
(Inv. 2) 


Three of the lessons were enactments of the same investigation (Investigation 2 on using doubling and 
halving to find equivalent ratios) while the fourth was an enactment of a similar investigation 
(Investigation 4 on generating equivalent ratios with discrete and continuous quantities). These four 
teachers had between six and 15 years of experience.* Ms. Aguila and Ms. Chancellor both had 
leadership roles in their schools (Math Lead and Department Chair, respectively). It was Ms. Elmore’s 
first year of teaching SunBay Math units. From the video records, a narrative of each lesson enactment 
was constructed, telling the story of the lesson from start to finish, highlighting each of the elements of 
structural and pedagogical integrity within the context of the specific lesson, and constructing rich 
descriptions with excerpts of teacher and student dialogue. 


As highlighted in the last chapter, teacher questioning was a high-leverage pedagogical practice closely 
associated with maintaining or lowering the cognitive demand of students’ opportunity to learn. We 
therefore further analyzed transcripts of the video recordings of these four lesson enactments to 
provide a more nuanced picture of the nature and role of teacher questions. All questions posed by the 
teacher were marked and further analyzed to better understand teacher questioning and students’ 
opportunities for mathematical engagement in these contrasting implementations of SunBay Math 
lessons. We adapted the methodology and typology of teacher questions in mathematics from Boaler 
and Brodie (2004) to code instances of teachers posing questions, either in whole class discussions or 
while students were working in groups or independently (see Table 8). Following Boaler and Brodie’s 
(2004) methods, we considered “utterances that had both the form and function of questions, and 
which were mathematical” (p. 777). Repeated questions during one teacher turn, even if altered slightly, 
were counted as one question. Each transcript was coded by two researchers and any discrepancies 


4 All teacher and student names are pseudonyms. 
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were resolved through discussion. The preponderance of these question types are reported as a 
percentage of the total questions asked in each case study and later summarized in the cross-case 
analysis. 


Table 8. Question Types from Boaler & Brodie (2004) 


Question Type Characteristics 


Gathering information Requires immediate answer, rehearses known facts/procedures, or enables 
students to state facts/procedures 


Inserting terminology Once ideas are under discussion, enables correct mathematical language to 
be used to talk about them 


Exploring mathematical meanings Points to underlying mathematical relationships and meanings;. Makes links 


and relationships between mathematical ideas and representations 

Probing Asks students to articulate, elaborate, or clarify ideas 

Generating discussion Solicits contributions from other members of class 

Linking and applying Points to relationships among mathematical ideas and mathematics and 


other areas of study/life 


Extending thinking Extends the situation under discussion to other situations where similar 
ideas may be used 


Orienting and focusing Helps students to focus on key elements or aspects of the situation in order 
to enable problem-solving 


Establishing context Talks about issues outside of math in order to enable links to be made with 
mathematics 


The True Colors Mural unit is comprised of 10 investigations, each designed to be completed in 45 to 50 
minutes, revolving around the real-world context of mixing paint for a company that makes murals. The 
unit begins with an introduction to ratio through the context of mixing red and yellow pips to make and 
compare different shades of orange. In Investigation 2, students use the technology to make equivalent 
blends (ratios) from black and white paint, explore the different representations (see Figure 1), and 
learn the symbolic notation for writing ratios. The tasks in Investigation 2 are constructed to allow 
students to discover that the numbers in the ratio can be doubled or halved to create an equivalent 
ratio, and that they can also build up ratios additively with the container to create equivalent ratios. At 
the end of the lesson, students are asked to generalize from the problems they have completed to 
explain how to create blends that are the same shade of grey. The investigation therefore offers 
students opportunities to identify patterns, make explicit connections between representations, 
strategies, and mathematical concepts, and form and justify generalizations based on this work without 
necessarily using a known procedure. Later, in Investigation 4, students have the opportunity to use 
what they have learned about equivalence and unit ratios to solve problems that involve both 
continuous and discrete quantities. 
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Ms. Chancellor: Students Working in Groups with High Pedagogical Integrity 

The enactment of True Colors Investigation 2 in Ms. Chancellor’s sixth-grade classroom provides an 
example of students working collaboratively in small groups while using dynamic mathematical 
representations in the software to make conceptual connections to the mathematics. 


The lesson was enacted with high structural integrity, closely matching what was described in the 
written curriculum, with all the suggested investigation parts as well as the appropriate balance of 
activity structures: whole class discussions facilitated by the teacher at the beginning and end; as well as 
an extended period of time for group work, where students worked collaboratively using the technology 
to solve problems at their own pace while the teacher circulated. 


The lesson was also enacted with high levels of pedagogical integrity. Ms. Chancellor’s interactions with 
students during small group work with the technology emphasized and reinforced the use of technology 
with an inquiry approach. While students were working on the tasks, she continually prompted them to 
use the different representations in the software to check their predictions, “see what happens,” and 
make connections. Rather than telling them if a solution was correct, she reminded them, “See if you 
can get it to line up in the software,” or “Test it and find out. See if it matches up.” 


Through her interactions with students, Ms. Chancellor reinforced the PCE cycle, encouraged 
exploration and the use of different representations in the technology, and reinforced norms for 
working collaboratively. She emphasized and defined the predict and check parts of the PCE cycles 
throughout the lesson. For example, before having students work in groups on the tasks, she gave the 
following directions to orient them around the PCE cycle: 


When it says predict, you make a guess. Don’t check it until you get to the check piece. 
You're not going to get in trouble if your prediction was wrong. [It’s] just to make you 
think about it before you start. 


Ms. Chancellor also oriented students towards collaboration around the mathematics. While they were 
working, she often reminded them to talk to their groupmates, at one point saying to the whole class, 
“Remember to use your groupmates as a resource,” and in another instance saying to a student, “Did 
you share that logic with your group members?” During the group work portion of the class, she also 
encouraged students to continue exploring mathematical relationships they were discovering. When 
students noticed the doubling or halving pattern, she responded either by saying, “Hmm, interesting...” 
or by redirecting them to share this observation with their group members. She answered procedural 
questions or questions about the software that students raised, but also posed questions back to them 
that required reasoning, for instance, “Why do you think it’s seven?” and “What do you notice about the 
original numbers and the new numbers?” Through these responses to students, she communicated the 
expectation that they should be reasoning about the tasks and the mathematical relationships with their 
groupmates. 


Ms. Chancellor posed high-level questions throughout the lesson. In the whole group portions of the 
lesson, the majority of the questions Ms. Chancellor posed (63%) were focused on gathering information 
but she also asked some questions that required students to explore meaning (15%) or probed their 
thinking (12%). However, once a student responded to these higher-level questions, she tended to 
either restate or expand on the explanation, or follow up with more directive questions. (In other words, 
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there was still room for improvement in terms of engaging students in high-level reasoning and 
explanation.) 


In sum, this enactment of Investigation 2 reflected collaborative small group work with teacher 
facilitation of opportunities for students to develop understanding of mathematical procedures 
(doubling and halving a ratio to create an equivalent ratio) that was connected to visual and dynamic 
mathematical representations in the technology. 


Ms. Deegan: Students Working in Groups with Low Pedagogical Integrity 

Like the case of Ms. Chancellor, the enactment of Investigation 2 in Ms. Deegan’s classroom reflected 
substantial time for students to work in groups buttressed by whole class discussions. However, the 
teacher practices in facilitating and supporting student work on the tasks led to a focus on completion 
rather than on exploration or connections. This case offers an example of how a teacher may transfer 
responsibility for learning to the students without providing adequate support to ensure that they 
engage with mathematical ideas at a high level of reasoning or sense-making. 


While the lesson was enacted with a high level of structural integrity to the balance of activity 
structures, it did not incorporate all investigation components: there was no Wrap-Up discussion. After 
having students finish the Problem Solving section from the previous lesson and completing the Warm- 
Up, Ms. Deegan began the Main portion of the investigation by reading the first task to the students. 
During the whole class introduction, she posed questions that elicited factual responses from students 
and then followed up on their responses only to confirm when the answer was correct. For the next half 
hour, students worked in their groups to answer the tasks using their iPads and workbooks. 


In terms of pedagocial integrity, Ms. Deegan’s facilitation of student work did not maintain a focus on 
using technology for exploration, engaging in PCE, or discussion of mathematical ideas. While students 
worked in groups, Ms. Deegan circulated and checked in with various groups to gauge and encourage 
their progress through the tasks, as well as to answer questions. She monitored student work by 
directing them to work through the tasks in the workbook, which meant that students worked on 
different parts of the PCE cycle that were embedded in the tasks. Unlike Ms. Chancellor, she did not 
remind the whole class what it meant to predict, check or explain. When a girl was asking for help with a 
predict question, Ms. Chancellor responded: “It’s what you think is going to happen. What do you think 
is going to happen? | don’t know. You got to tell—they want you to do it.” Similarly, when a student 
asked for help on an explain question by asking, “Can | just say | know because | compared them to the 
other grays?”, Ms. Deegan confirmed that her response was fine as long as it was her explanation rather 
than push her to focus on the mathematical relationships. In this way, she was encouraging student 
autonomy but not providing support or direction to help them focus on inquiry or on the important 
mathematical ideas. 


Ms. Deegan circulated while students were working in groups, but her interactions focused almost 
entirely on logistics and completion rather than on using technology to explore and reason about 
mathematical relationships. When students asked questions, she responded by reading the problems 
aloud or by directing them to the next step, which often involved using the dynamic representations: 
“So now, what does it say now? Keep reading the next part. You guys make the container yet?” As she 
circulated, her comments focused on checking for progress on the workbook page rather than looking at 
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the quality or reasonableness of their answers. She continually urged them to keep moving through the 
problems: “You have two more pages to go actually. If you get this done, you’ll be ahead of the game.” 


While students were working in groups, Ms. Deegan occasionally asked for confirmation that they 
understood the task, but then did not press further: “How are you guys doing over here? 
Understanding? Ok good.” A few minutes before the end of the period, she told students to finish up the 
question they were working on and turn in their iPads. There was no time spent working on or 
discussing the Wrap-Up to Investigation 2. 


In sum, this implementation represented a case of a teacher providing a lot of time for students to work 
together and with a great deal of autonomy. Students may or may not have engaged in mathematical 
reasoning, making connections, or exploring meaning while they were working in their groups with the 
technology, but their interactions with the teacher were almost exclusively focused on procedural 
logistics and completion. The fact that there were no instances of whole class discussion around the 
important mathematical ideas of the investigation makes it difficult to ascertain whether all students did 
indeed have opportunities to engage in meaning-making or develop an understanding of the 
mathematical ideas embedded in the tasks. 


Ms. Aguila: Whole Group Teacher Facilitation with High Pedagogical Integrity 

The same lesson was enacted in Ms. Aguila’s sixth-grade classroom with more whole group teacher 
facilitation than what was recommended by the SunBay Math curriculum, reflecting a lower level of 
structural integrity. However, this enactment received the highest overall rating for level of 
implementation and teacher questioning. The case study highlights several important teacher practices 
that maintained and complimented the pedagogical integrity of the lesson as intended. 


During both the Problem-Solving (homework) review and Warm-Up, she consistently called on several 
students for each problem and followed up with probing questions, such as “What did you do?” and 
“Why does that work?” Often when a student would give a response she would ask the rest of the class 
if they agreed and would then call on another student to explain why. She ended the Warm-Up by 
asking the class to look for patterns across the problems and eliciting several responses. This was not a 
suggestion in the teacher guide, but it emphasized the importance of doubling and halving, a main focus 
of this investigation. 


During the Main portion of the investigation, Ms. Aguila led the class through each task: typically, by 
having a student read the question aloud, then clarifying the task, giving students a few minutes to work 
together on the iPad, asking a student to share or come up and demonstrate the solution to the class, 
and finally engaging the class in further demonstration or discussion. In this way, the class moved 
through all four of the tasks in the Main section at the same pace, and the solution to each question was 
discussed as a class before moving on to a new question. Despite the teacher exerting control over the 
pace and direction of the work, there were several elements that supported a high level of cognitive 
demand throughout the lesson. 


First, Ms. Aguila engaged students in complete cycles of PCE, emphasizing the meaning of prediction 
(“There’s no right or wrong answer—you’re predicting”) and eliciting multiple answers, correct and 
incorrect. She then had students check their responses with the technology-based tools (e.g., “Check 
the spectrum—were both triangles on the same spot?”). Once the solution had been verified with the 
software and demonstrated, she then asked multiple students to verbalize an explanation before having 
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them write one down in their workbooks. She also provided support for how students should explain by 
drawing on the technology without telling them exactly what to write: “What strategy did you use? How 
are those numbers related? Did you use the container? Did you use the spectrum? Write down how you 
know.” 


Throughout the lesson, Ms. Aguila prompted students to use different representations when checking 
their predictions with the technology. She encouraged them to click on the artwork, spectrum, 
container, and mixer to check if the blends they created were equivalent. During whole class 
discussions, she posed questions about dynamic demonstrations of different mathematical 
representations. She also constructed an organized list of equivalent ratios on the board, which she 
continually added to and referenced throughout the lesson to highlight the multiplicative relationships 
between the quantities. In addition, Ms. Aguila introduced some of her own questions and a comparison 
to another familiar context— mixing chocolate milk—to enhance student understanding. While Ms. 
Aguila’s use of written representations on the board and the introduction of familiar contexts were 
adaptations of the SunBay Math materials, these adaptations maintained or enhanced the cognitive 
demand of the tasks and reflected the overall mathematical goals of the lesson as written. 


Ms. Aguila asked a variety of question types throughout the lesson, including questions that focused on 
mathematical meanings and relationships (Boaler & Brodie, 2004). She frequently asked questions that 
generated student thinking and reasoning, such as: “How come that works? What do the numbers tell 
you? What was your strategy? Is there a pattern? Why do you think? How do you know they are the 
same color?” Finally, Ms. Aguila elicited multiple responses for each question, even after the correct 
answer had been provided. This is something we rarely saw in other video-recorded lessons. In this 
class, incorrect answers were both elicited and probed. 


In sum, the lesson enactment by Ms. Aguila represents an implementation that maintained a high level 
of cognitive demand not only by following the intended curriculum but, in particular, through the 
questioning and pedagogical strategies that she brought to the lesson. It reflected her deliberate use of 
the resources in the curriculum along with her own pedagogical moves and orientation, to construct a 
dialogic learning experience where students engaged in reasoning around the important mathematical 
ideas. 


Ms. Elmore: Whole Group Teacher Facilitation with Low Pedagogical Integrity 

While the last case is an enactment of Investigation 4 rather than Investigation 2, it offers an important 
contrast in that the teacher practices focused more directly on procedural knowledge and solutions, 
something that was common in the larger sample of lesson enactments. The enactment of this lesson 
had low structural integrity in terms of both lesson components (having neither a connected Warm-Up 
nor a Wrap-Up) as well as the balance of activity structures (entirely whole group), along with low 
pedagogical integrity across all dimensions. This enactment reflected significant adaptations to the 
lesson, but in a way that detracted from the mathematical goals and the integrity of the lesson as 
designed. 


In Investigation 4, students are expected to use what they have learned about ratio (including unit 
ratios) to generate equivalent ratios, find missing values and use table and graph representations. The 
tasks are designed to be open to multiple strategies. However, Ms. Elmore spent the first 15 minutes of 
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class having students complete and then review an exit ticket®, while highlighting how to find the unit 
rate through division. In this way, the lesson began with a procedural focus on the steps of a specific 
strategy before students had any opportunity to explore relationships between proportional quantities 
with the technology. 


The first task in the investigation involves using the dynamic mathematical representations in the 
technology to complete a table of equivalent ratios for making different amounts of clay with a given 
ratio of 3 cups of flour to 1 cup of salt. After reading the introduction and task aloud, Ms. Elmore 
underlined parts of the task and demonstrated how to write the given ratio. She then told them to enter 
the given ratio into the mixer tool while she did the same on the projection screen, and asked students 
to figure out how many white pips they needed to create the same ratio with 12 black pips. Very quickly 
a student offered that it would be 3 white pips, explaining, “I divided 12 by 3 and | got 4.” Ms. Elmore 
restated and expanded on this solution strategy by highlighting the calculations and recording them on 
the projected version of the workbook. 


She then put the solution (12 black pips and 4 white pips) into the mixer tool in the technology and 
asked students if it was correct, emphasizing how to make sure the spectrum marker was lined up. In 
this way, the technology was used to check the answer found through division. She then asked the 
students how else they could check to make sure the blends were equivalent. When two students 
responded by restating the multiplicative relationship between the numbers, Ms. Elmore suggested: 
“Could we check the container?” However, rather than have students explore or demonstrate what the 
container could show, she answered her own question by stating: “Yes. Ok, that was another way.” 


The next entry in the table required determining how many cups of salt would be used for 1 cup of flour. 
Students struggled to provide the correct answer to this question, first answering 12, then 1, then 3, and 
finally 6. Rather than allow students to explore any of these predictions on the software, Ms. Elmore 
asked them to look for a pattern in the numbers and again led them to the procedure of dividing by 3 
through more direct questioning while she recorded it on the projected screen: 


Ms. Elmore: You guys could check those responses with the iPad and see if those blends are 
equal, but | want you to think about if there is a relationship between the flour 
and salt? Just the first row. How do you think they got from 12 to 4? How do 
you think they got from 3 to 1? 


Student: Divided by 3. 


Ms. Elmore: Divided by 3. That could work! 12 divided by 3 is 4. And 3 divided 3 is 1, so if you 
continue that pattern it should be 1 divided by 3, am | right or wrong? 


Instead of having students finish the table (“We'll fill it in later’), Ms. Elmore moved on to the next task 
which involved using the graphing tool. Ms. Elmore first reviewed parts of the linked graph using the 
projected display and then explained how each point represented the ratios of 3 to 1 and 12 to 4 that 
had been entered. As the class returned to the question in the workbook, she said, “Explain what each 
point means.” Ms. Elmore told students to note what she was writing on the screen, “We are going to 


5 Exit tickets were created by one of the districts to complement the units with formative assessment 
opportunities. 
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put ‘each point shows one of the’—please write this down—'clay blends ratios from the table above.” 
For students, answering the explanation question consisted of copying down the teacher’s response. 


At the end of the lesson Ms. Elmore directed students to hit the button marked “lines” and then posed 
and answered her own question: 


What happened? Do you see the difference now? That’s another way to check that 
your ratios are equal. Ok? The fact that both of these points are on the same line, 
they’re showing they have the same blend of 3 blacks to 1 white ratio. 3 blacks to 1 
white ratio. So they share the same unit ratio and they create the same color blend. 


As these examples show, there was mention and some demonstration of the different 
representations—the spectrum, the container, and the graph. However, the focus was on 
demonstration of the visual representations as confirmation of the procedure or for verification of the 
answer, rather than to test out predictions, explore, or look for relationships. 


While Ms. Elmore posed some of the questions that were designated as “predict” or “explain” from the 
guide, the overall focus was on using the numbers and mathematical procedures to solve the tasks and 
writing down the correct explanation. The majority of questions (63%) focused on gathering 
information. While she did ask some questions that focused on exploring meaning, in every case she 
followed up with more directive questions that funneled directly to a numerical answer or one she 
provided herself. Overall, this lesson enactment represented a teacher-directed lesson centered on 
correctness and procedural knowledge. Ms Elmore provided direct instruction around a specific method 
(using division to find a unit rate) before having students work on the tasks, and answered some of the 
higher-level questions herself, only asking students to provide numerical answers. In turning the focus 
away from mathematical exploration towards the application of a procedure, she significantly lowered 
the cognitive demand of the task for students. 


Cross Case Analysis: Unpacking Pedagogical Integrity 

These four lesson enactments illustrate differences in the ways that the written curriculum is 
transformed through the pedagogical integrity of specific teaching practices and interactions between 
teacher and students in the classroom. First, we study these cases for the variation in the specific 
elements that are unique to SunBay Math lessons—PCE cycles and the use of the interactive technology- 
based tools to develop representational fluency (Zbiek et al., 2007). We then explore teacher 
questioning and student discussion in terms of more nuanced discourse practices. Finally we examine 
different teacher orientations that can be seen in these cases, specifically in terms of the role they 
adopted as a teacher and their stance towards the use of the curriculum units. 


SunBay Math Practices 

All four lessons incorporated opportunities for students to engage in the parts of the PCE cycle 
embedded within the tasks in the curriculum. Ms. Aguila continually reinforced norms for what it meant 
to predict, check and explain as she taught the lesson. Ms. Chancellor also defined what it meant to 
predict, and in her interactions with small groups indirectly emphasized the inquiry stance embodied in 
PCE by posing questions to students (“What do you notice?”; “Hmmm, interesting...”). In both cases, 
high levels of integrity to PCE provided important guidance for students in how they should interact with 
the tasks and representations in the technology. Ms. Deegan, on the other hand, referenced the 
explanation aspect of PCE as what “they” wanted you to do and stated that as long as it was the 
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students’ own explanation, it was acceptable (“This is your thoughts”). Her overall focus on PCE was 
more on completing each of the questions than on engaging in inquiry. Ms. Elmore did not directly 
address the meaning of PCE. Since she was teaching Investigation 4 it may be that students were already 
familiar with this routine. However, by walking her students through procedures for solving the 
problems before using the software, she in effect circumvented the PCE inquiry cycle. In both of these 
cases, the low levels of integrity to PCE meant that while students were using the technology, there was 
no guidance as to how they should be engaging with the dynamic mathematical representations to 
answer the questions. 


In all four lesson implementations, students had some opportunities to use the software to make 
connections and develop understanding. However, Ms. Aguila and Ms. Chancellor directly supported the 
students’ use of technology for exploration and developing representational fluency by prompting them 
to use different representations to check their predictions and look for relationships. Both teachers 
posed questions to focus students on using the technology to reason about the mathematics. Ms. Aguila 
also demonstrated these relationships to the whole class by using the representations dynamically (i.e., 
clicking between two blends and switching back and forth between the spectrum and the container to 
highlight mathematical relationships visually). It was unclear in Ms. Deegan’s and Ms. Elmore’s lessons 
whether students were making these connections through their use of the technology. In Ms. Elmore’s 
class those opportunities were limited by focus on the use of representations to check the veracity of 
answers found through procedures. Table 9 summarizes the variation in both PCE and the approach to 
technology use across all four cases. 


Table 9. SunBay Math Practices Across Cases 


SunBay Math Practice Ms. Aguila Ms. Chancellor Ms. Deegan Ms. Elmore 
PCE Reinforcement of Reinforcement of Completion Circumvention 
meaning meaning and 
redirection 
Approach to Demonstration, Exploration and Use without Demonstration and 
Technology Use exploration and supporting guidance procedural use 
supporting representational 
representational fluency 
fluency 


Discourse Practices 

The more in-depth analysis of teacher questioning in these cases yields dramatic differences in the 
prevalence of different types of questions across the four enactments. Table 10 illustrates the six most 
common types of questions in order of prevalence across the four cases. As Table 10 shows, Ms. Aguila 
and Ms. Chancellor asked a lot more questions and posed more probing questions and more questions 
that focused on exploring meaning. Gathering information comprised the vast majority of questions in 
Ms. Deegan’s lesson and most of the questions in Ms. Chancellor and Ms. Elmore’s lessons. Ms. Aguila’s 
lesson had nearly an equal balance between gathering information and probing and was the only case 
where there were any questions in the categories extending thinking or linking and applying. 
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Table 10. Types of Questions Posed in Lesson Enactment Case Studies (Boaler & Brodie, 2004) 


Teacher Total no. Gathering Probing Exploring Generating Inserting Other® 
of meaning Discussion terminology 
Questions 
Aguila 93 37(40%) 32(34%) 11 (12%) 8 (9%) 3 (3%) 2 (2%) 
Chancellor 124 79 (63%) 15 (12%) 19 (15%) 2(2%) 4 (3%) 5 (4%) 
Deegan 37 33 (89%) 1 (3%) 1 (3%) 0 2 (5%) 0 
Elmore 41 26 (63%) 3 (7%) 8 (20%) 3 (7%) 1 (2%) 0 


The way in which teachers responded to incorrect student responses also varied in these lesson 
enactments, reflecting different opportunities for students to engage in productive struggle. Ms. Aguila 
and Ms. Elmore both elicited incorrect responses during whole class discussions, but Ms. Aguila elicited 
additional responses even after a student offered the correct response, thereby ensuring incorrect 
responses would get out into the discussion. Ms. Elmore elicited them only when the first student she 
called upon gave an incorrect response. More importantly, once elicited, Ms. Aguila probed the 
student’s thinking, thereby communicating confidence in students’ ability to reason about mathematics. 
In contrast, Ms. Elmore either did not acknowledge an incorrect response or redirected the student to 
the correct procedure and response. Both moves fail to signal confidence in students’ capacity to reason 
through the problem. While Ms. Chancellor did not elicit any incorrect responses during whole group 
discussion, the way she responded to questions while students were working in groups encouraged 
them to figure it out together. This emphasis not only helped to maintain the opportunity for productive 
struggle but also encouraged collaboration among the group. Table 11 summarizes these four discourse 
practices across the cases. 


5 Orienting and focusing, extending thinking, and linking and applying. 
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Table 11. Summary of Discourse Practices Across Cases 


Discourse Practice Ms. Aguila Ms. Chancellor Ms. Deegan Ms. Elmore 
Questioning High level, varied, High level, focused Low level— Low level- 
and focused on on mathematical encouragement and _ procedural 
mathematical meaning and direction 
meaning and relationships 
relationships 
Student Multiple strategies Single strategy None Single strategy 
explanations 
Response to student Restated, expanded Expanded on, follow N/A Redirected toward 


explanations 


Incorrect responses 


on, prompted for 
more 
Elicited and probed 


up with more 
directive questions 
Orients to group 


Leads to correct 
response 


desired response 


Ignores or leads to 
correct response 


Teacher Orientation 

The questions teachers asked and the way in which they responded to student contributions also 
illustrate differences in the role of the teacher in facilitating student learning. Ms. Aguila’s orientation as 
a facilitator was to elicit student’s ideas and then build on those ideas (by probing, inserting examples, 
or focusing them on patterns) to help them make connections to the mathematics. Ms. Chancellor 
continually prompted students to make sense of the mathematics while they were working 
collaboratively on the tasks. She took a less active role, however, in pressing on or connecting their ideas 
during whole class discussions. Ms. Deegan took on the role of a manager during whole class 
discussions. While students worked through the tasks, she made sure they were moving through each 
question and demonstrated correct responses, but she did not address the quality of their explanations. 
Ms. Elmore took the most directive approach, modeling strategies for solving problems, highlighting 
vocabulary and written formats, and modeling explanations. 


These four cases also illustrate significant differences in the orientation of the teachers towards the use 
of the curriculum. Ms. Chancellor followed the curriculum with the most integrity, both in terms of 
structure and pedagogy. Her lesson followed the recommended timing for the parts of the 
investigations and the balance of activity structures. She emphasized PCE cycles and oriented students 
towards collaboration and productive struggle. 


The enactment in Ms. Aguila’s class, which had low structural integrity with high pedagogical integrity, 
however, suggests that structural integrity is less important for overall implementation. This is further 
supported by the fact that the enactment of Ms. Deegan’s lesson reflected structural integrity without 
pedagogical integrity; while students completed the tasks as written in the curriculum, her role as a 
teacher—encouraging autonomy with the goal of completion—only partially reflected the goals of 
SunBay Math. As a result, the opportunity for students to engage in productive struggle around 
important mathematical ideas was diminished. 


Ms. Aguila adapted the lesson in terms of both structure and pedagogy, but these adaptations were in 
line with the goals of SunBay Math and in many cases enhanced the lesson as written (e.g., eliciting and 
probing correct and incorrect responses, introducing her own representations and examples, and posing 
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a variety of question types). Ms. Aguila demonstrated a high degree of what Brown (2009) defines as 
pedagogical design capacity, or "skill in perceiving the affordances of the materials and making decisions 
about how to use them to craft instructional episodes that achieve her goals" (p. 29). 


Ms. Elmore also adapted the lesson in ways that reflected her goals, but her goals were to ensure that 
students learned correct procedures and obtained correct answers, instead of encouraging them to 
engage in productive struggle around important mathematics. Her practices reflected a procedural 
orientation towards mathematics learning and teaching. This may have been a result of having less 
opportunity for professional support as it was her first year of implementing SunBay Math. Importantly, 
the practices of Ms. Deegan and Ms. Elmore when teaching these SunBay Math lessons did not produce 
evidence of students’ capabilities to engage in high-level reasoning and sense-making. For some 
teachers implementing SunBay Math, completion of all the tasks in the order they are presented may 
take precedence over ensuring that each lesson had a coherent beginning, middle and end around the 
instructional goal. Table 12 summarizes the orientations to teacher role and curriculum use across the 
cases. 


Table 12. Summary of Teacher Orientations Across Cases 


Discourse Practice Ms. Aguila Ms. Chancellor Ms. Deegan Ms. Elmore 
Teacher role Eliciting, building on, Prompting student Management Modeling correct 
and connecting meaning making without connections strategy and 
student ideas and knowledge 
mathematics 
Use of Curriculum Adapting Offloading Offloading and Improvising and 
adapting adapting 


Finally, Table 13 combines the teacher practices and orientations discussed in these case studies. 
Looking at the practices within each case, one can see a thread that links them into an overall 
instructional approach: Ms. Aguila’s practices focus on engaging students in high-level reasoning, while 
Ms. Chancellor’s focus on facilitating collaborative reasoning. Ms. Deegan’s practices focus on 
completion, while Ms. Elmore’s focus on procedural knowledge and correctness. 
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Table 13. SunBay Math Practices, Discourse Practices, and Teacher Orientations Across Cases 


more 


questions 


Ms. Aguila Ms. Chancellor Ms. Deegan Ms. Elmore 
PCE Reinforcement of | Reinforcement of | Completion Circumvention 
= meaning meaning and 
& ¥ redirection 
= s Approach to | Demonstration, Exploration and Use without Demonstration 
ao © Technology | exploration and supporting guidance and procedural 
2 Se Use supporting representational use 
representational fluency 
fluency 
Questioning | High level, varied, | High level, Low level— Low level— 
and focused on focused on encouragement procedural 
mathematical mathematical and direction 
meaning and meaning and 
” relationships relationships 
s Student Multiple Single strategy None Single strategy 
v explanations | strategies 
5 Response to | Restated, Expanded on, N/A Redirected 
8 student expanded on, follow up with toward desired 
a explanations | prompted for more directive response 


TeacherOrientatio 
n 


Incorrect Elicited and Orients to group Leads to correct Ignores or leads 
responses probed response to correct 
response 
Teacher role | Eliciting, building Prompting Management Modeling correct 
on, and student meaning without strategy and 


connecting 
student ideas and 
mathematics 


making 


connections 


knowledge 


Use of 
curriculum 


Adapting 


Offloading 


Offloading and 
adapting 


Improvising and 
adapting 
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Chapter 5: Discussion and Implications 


The results of this study suggest that elements of structural integrity—giving students access to 
technological tools and having students work in groups to complete the tasks as sequenced in the 
curriculum—is not sufficient to ensure opportunity to learn in the ways intended by the curriculum 
designers. 


Developing curricular materials that can support and enhance students’ opportunity to learn is a 
laudable and important goal. But while it may seem that activities embedded in technology remove, at 
least to some degree, the effect of the teacher, our study found significant variation in teacher 
instructional practices that had important consequences. Because of the way the teacher presented and 
orchestrated discussion around the tasks, the technology was either used procedurally (e.g., to find 
answers or check previously determined answers), used for exploration but without any guidance, or 
used for exploration along with reasoning, explanation and connections to important mathematical 
ideas. Just like more traditional text-based curricula, technology is adapted in practice and can be 
transformed by teachers and students. These findings highlight the importance of the technological 
pedagogical integrity of an enacted lesson, or how teachers frame, make use of, and support the 
purpose of the technological tools and/or what students should be doing with them. 


The presence and nature of whole group introductions and conclusions to bookend the work around the 
task is equally important for students’ opportunity to learn. A limitation of the study is that we are not 
able to ascertain how individual students or groups of students were discussing or making sense of the 
activities. Yet this limitation also points to a larger issue: what students learn in group work is likely 
incomplete and relatively unsophisticated, but it is through whole class discussion that the teacher can 
elicit, guide and consolidate student thinking in relation to the important mathematical ideas (Sleep, 
2012; Stein, Engle, Smith, & Hughes, 2008). In the absence of high quality discussion focused on the 
important mathematical ideas, the learning that results is more or less dependent on the strengths and 
weaknesses of individual students and interactions that may or may not occur with the teacher. 


Finally, the findings suggest that teachers’ repertoires of instructional practices influence their ability to 
use and adapt the curriculum in ways that reflect the principles of teaching and learning of SunBay 
Math. All teachers adapted the curriculum in some way, albeit to different degrees and in relation to 
different dimensions (timing, activity structure, PCE cycles, use of technology, or explanation and 
discussion). In some cases, these adaptations did not disrupt the overall emphasis on engaging students 
in high-level reasoning around mathematical concepts and ideas. In other cases, they enhanced it. 
However, in many cases teacher adaptations subverted the design and intent of the materials. Teachers 
will always adapt curriculum to some degree. The question for reform efforts based on curriculum as a 
vehicle for change then becomes how to build teacher capacity for decision-making around technology- 
based curriculum that is responsive to the particular context, but still aligns with the curriculum 
developers’ intent and/or district priorities. 


Being able to use and adapt curricular materials to construct a lesson that coheres around the learning 
goals requires what Brown (2009) calls pedagogical design capacity. For example, although Ms. Aguila 
did not get to the discussion of the Wrap-Up question in the written curriculum, she had students share 
strategies for the last task in a way that accomplished the same goals. In this way, Ms. Aguila 
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demonstrated her pedagogical design capacity by presenting a coherent lesson with clear learning goals. 
Similarly, while she directed students through the technology-based tasks in a more teacher-directed 
way than was suggested by curriculum design, she orchestrated high quality discussion linked to the 
learning goals throughout the lesson. Whether or not teachers make adaptations to the curriculum 
seems less important, then, than whether or not they have the capacity to make changes in response to 
local context that keep the overall intent of the curriculum design intact. As Taylor (2013) argues, rather 
than looking for “teacher-proof curriculum,” we might be better off developing “curriculum-proof 
teachers” who can use and adapt curriculum in ways that expand learning opportunities for students. 


Implications 

New learning technologies are being promoted as a vehicle for instructional reform in mathematics, and 
evidence suggests that technology can be used to make learning more student-centered, increase the 
authenticity of mathematical activity, provide feedback that promotes reflection, and shift the authority 
for determining mathematical truth from the teacher or text to the students (Heid, 1997). The idea that 
technology can be a catalyst for these shifts, however, rests on the assumption that teachers have the 
capacity—beliefs, knowledge, and practices—to support and orchestrate the use of technological tools 
for meaningful mathematics instruction. 


As a set of replacement units, SunBay Math is designed to position students as active learners and users 
of technology. The curriculum itself helps to orchestrate this positioning both in the way the technology 
operates and in the way the tasks and investigations are structured. However, this study emphasizes the 
central role teachers play in enacting the curriculum and key areas of leverage for SunBay Math 
professional development and ongoing support for high-level implementation: teacher capacity and 
instructional practices. 


The appropriation of new pedagogical practices (Grossman, Valencia, Evans, Thompson, Martin, & Place, 
2000) often involves undoing old practices—not just taking on new ones—and these old practices may 
be deeply entrenched (Munter & Correnti, 2017). In order to shift the work of problem-solving and 
“figuring it out” to students, teachers must organize the classroom and lesson structure differently, 
establish new norms and expectations, use different discourse moves to elicit student thinking, support 
student thinking and reasoning around challenging tasks, and expertly weave together multiple 
strategies and levels of understanding towards the learning goal. Supporting students to make 
connections to mathematical ideas through dynamic mathematical representations in the technology, 
adds another layer of complexity. 


These practices were evident to some degree in the higher-level lesson enactments, but in lower-level 
enactments we saw students work in groups with the technology without guidance or direction, 
teachers asking questions that focused on gathering information and then following up with more 
directive questions to steer students towards the correct answer or procedure, or teachers 
demonstrating a procedure or method before having students use the technology to complete practice 
problems. These practices are common in more traditional and didactic models of math instruction, and 
in fact, had been promoted by both districts in recent years. It is not surprising then that we saw these 
patterns persist in the second year of SunBay Math implementation. Learning and adopting new 
practices takes time, and occurs at different rates for different teachers (Grossman et al., 2000; 
Desimone, Porter, Garet, Yoon,& Birman, 2002). While expectations change, and new curricula is 
provided, teachers are rarely given adequate time or opportunities to develop, practice and refine new 
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instructional practices in their own classroom settings. This study highlights that these instructional 
practices are essential for providing high quality learning experiences for students with technology. 
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Appendix A. lQA Rubrics (Boston, 2012) 


Potential of the Task Rubric 


Score 


Description of Score Level 


The task has the potential to engage students in exploring and understanding the nature of mathematical 
concepts, procedures, and/or relationships, such as: 


© Doing mathematics: using complex and non-algorithmic thinking (i.e., there is not a predictable, well- 
rehearsed approach or pathway explicitly suggested by the task, task instructions, or a worked-out example); 
OR 


e Procedures with connections: applying a broad general procedure that remains closely connected to 
mathematical concepts. 

The task must explicitly prompt for evidence of students’ reasoning and understanding. 

For example, the task MAY require students to: 

solve a genuine, challenging problem for which students’ reasoning is evident in their work on the task; 

develop an explanation for why formulas or procedures work; 

identify patterns and form and justify generalizations based on these patterns; 

make conjectures and support conclusions with mathematical evidence; 

make explicit connections between representations, strategies, or mathematical concepts and procedures; 

follow a prescribed procedure in order to explain/illustrate a mathematical concept, process, or relationship. 


The task has the potential to engage students in complex thinking or in creating meaning for mathematical 
concepts, procedures, and/or relationships. However, the task does not warrant a “4” because: 


e the task does not explicitly prompt for evidence of students’ reasoning and understanding; 

e students may be asked to engage in doing mathematics or procedures with connections, but the underlying 
mathematics in the task is not appropriate for the specific group of students (i.e., too easy or too hard to 
promote engagement with high-level cognitive demands); 

e students may need to identify patterns but are not pressed for generalizations or justification; 

e students may be asked to use multiple strategies or representations but the task does not explicitly prompt 
students to develop connections between them; 

e students may be asked to make conjectures but are not asked to provide mathematical evidence or 
explanations to support conclusions. 


The potential of the task is limited to engaging students in using a procedure that is either specifically called for or 
its use is evident based on prior instruction, experience, or placement of the task. There is little ambiguity about 
what needs to be done and how to do it. The task does not require students to make connections to the concepts 
or meaning underlying the procedure being used. Focus of the task appears to be on producing correct answers 
rather than developing mathematical understanding (e.g., applying a specific problem-solving strategy, 
practicing a computational algorithm). 


OR The task does not require student to engage in cognitively challenging work; the task is easy to solve. 


The potential of the task is limited to engaging students in memorizing or reproducing facts, rules, formulae, or 
definitions. The task does not require students to make connections to the concepts or meaning that underlie the 
facts, rules, formulae, or definitions being memorized or reproduced. 


Students did not engage in a mathematical activity. 
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Implementation of the Task Rubric 


Score | Description of Score Level 

4 Students engaged in exploring and understanding the nature of mathematical concepts, procedures, and/ 

relationships, such as: 

e Doing mathematics: using complex and non-algorithmic thinking (i.e., there is not a predictable, well- 
rehearsed approach or pathway explicitly suggested by the task, task instructions, or a worked-out 
example); OR 

e Procedures with connections: applying a broad general procedure that remains closely connected to 
mathematical concepts. 

There is explicit evidence of students’ reasoning and understanding. For example, students may have: 

e solved a genuine, challenging problem for which students’ reasoning is evident in their work on the 
task; 

e developed an explanation for why formulas or procedures work; 

e identified patterns and formed generalizations based on these patterns; 

e made conjectures and supported conclusions with mathematical evidence; 

e made explicit connections between representations, strategies, or mathematical concepts and 
procedures; 

e followed a prescribed procedure in order to explain/illustrate a mathematical concept, process, or 
relationship. 

3 Students engaged in complex thinking or in creating meaning for mathematical concepts, procedures, 
and/or relationships. However, the implementation does not warrant a “4” because: 

e = there is no explicit evidence of students’ reasoning and understanding; 

e students engaged in doing mathematics or procedures with connections, but the underlying 
mathematics in the task was not appropriate for the specific group of students (i.e., too easy or too 
hard to sustain engagement with high-level cognitive demands); 

e students identified patterns but did not make generalizations; 

e students used multiple strategies or representations but connections between different 
strategies/representations were not explicitly evident; 

e students made conjectures but did not provide mathematical evidence or explanations to support 
conclusions. 

2 Students engaged in using a procedure that was either specifically called for or its use was evident based 
on prior instruction, experience, or placement of the task. There was little ambiguity about what 
needed to be done and how to do it. Students did not make connections to the concepts or meaning 
underlying the procedure being used. Focus of the implementation appears to be on producing correct 
answers rather than developing mathematical understanding (e.g., applying a specific problem solving 
strategy, practicing a computational algorithm). 

OR There is evidence that the mathematical content of the task is at least 2 grade-levels below the grade 

of the students in the class. 

1 Students engage in memorizing or reproducing facts, rules, formulae, or definitions. Students do not 
make connections to the concepts or meaning that underlie the facts, rules, formulae, or definitions 
being memorized or reproduced. 

0 The students did not engage in mathematical activity. 
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Rigor of Teacher Questions Rubric 


4 


The teacher consistently asks academically relevant questions that provide opportunities for students 
to elaborate and explain their mathematical work and thinking (probing, generating discussion), 
identify and describe the important mathematical ideas in the lesson, or make connections between 
ideas, representations, or strategies (exploring mathematical meanings and relationships). 


At least two times during the lesson, the teacher asks academically relevant questions (probing, 
generating discussion, exploring mathematical meanings and relationships). 


There are one or more superficial, trivial, or formulaic efforts to ask academically relevant questions, 
probing, generating discussion, exploring mathematical meanings and relationships) (i.e., every 
student is asked the same question or set of questions) or to ask students to explain their reasoning. 
OR only one strong effort is made to ask academically relevant questions. 


The teacher asks procedural or factual questions that elicit mathematical facts or procedure or require 
brief, single-word responses. 


The teacher did not ask questions during the lesson, or the teacher’s questions were not relevant to 
the mathematics in the lesson. 


Student Discussion Following Task Rubric 


Students show/describe written work for solving a task and/or engage in a discussion of the important 
mathematical ideas in the task. During the discussion, students provide complete and thorough 
explanations of why their strategy, idea, or procedure is valid; students explain why their strategy works 
and/or is appropriate for the problem; students make connections to the underlying mathematical ideas 
(e.g., “I divided because we needed equal groups”). 

OR 

Students show/discuss more than one strategy or representation for solving the task, and provide 
explanations of why the different strategies/representations were used to solve the task. 


Students show/describe written work for solving a task and/or engage in a discussion of the important 
mathematical ideas in the task. During the discussion, students provide explanations of why their 
strategy, idea, or procedure is valid and/or students begin to make connections BUT the explanations 
and connections are not complete and thorough (e.g., student responses often require extended press 
from the teacher, are incomplete, lack precision, or fall short on making explicit connections). 

OR 

Students show/discuss more than one strategy or representation for solving the task, and provide 
explanations of how the different strategies/representations were used to solve the task but do not 
explain why they were used. 


Students show/describe written work for solving the task (e.g., the steps for a multiplication problem, 
finding an average, or solving an equation; what they did first, second, etc.) but do not engage in a 
discussion of why their strategies, procedures, or mathematical ideas work. [There are presentations of 
students’ work but no discussion. ] 

OR 

Students show/discuss only one strategy or representation for solving the task. 


Students provide brief or one-word answers (e.g., fill in blanks); 
OR 
Student’s responses are non-mathematical. 


There was no discussion of the task. 
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Appendix B. SunBay Math Enactment Rubrics 


Lesson Structure Rubric 


3 components 


Warm-Up, Main, Wrap-Up from one investigation occur in one class period 


2 components 


At least 2 of the 3 components from the same investigation are present 


1 component 


Only 1 of 3 components from the investigation are present 


None 


Lesson does not incorporate any of the three SunBay Math components 


Balance of Activity Structures Rubric 


Mostly Whole Lesson is mostly whole group and teacher-facilitated with some opportunities 

Group for students to work in groups 

Mostly Small Lesson has some teacher-facilitated whole group portions with extended 

Group opportunities for students to work in groups 

Whole Group Lesson is almost entirely whole group teacher-led with no opportunities for 
students to work in groups 

Small Group Lesson is almost entirely students working in groups while the teacher circulates 


Technology Use Rubric 


Exploration with 
connections 


Student exploration and use with the teacher helping to make connections to 
the mathematics 


Use without 
guidance 


Opportunity for exploration without guidance, connections, or explanation from 
the teacher; purpose or evidence of student use is more or less unclear 


Procedural use 


Technology is used in a procedural way (e.g., to check answers); technology is 
not used in a way that fosters student exploration or engagement in reasoning 
about the mathematical content 


No use 


Lesson does not incorporate any technology; or it is unclear the extent to which 
it was used 


Predict-Check-Explain (PCE) Rubric 


Consistent PCE 


Lesson incorporates all three aspects of the cycle: there are opportunities for 
students to predict, check, and explain and students engage in multiple PCE 
cycles while using the technology. 
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One PCE 


There is at least one complete PCE cycle enacted in the lesson, either in whole 
group or paired work, but PCE cycle is not consistent across the lesson. 


Partial PCE 


Two connected aspects of the PCE cycle are incorporated (e.g., students predict 
and check, but never explain) 


No PCE 


Lesson does not reflect any aspects of the PCE cycle 
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Appendix C. Task Implementation Scores in Relation to Potential of the 
Task on the IQA 


4 


3 

& Potential of the Task as Written 

@ High Level Implementation of the Task 
2 w Low Level Implementation of the Task 


1 
12345 6/7 8 9 1011121314151617181920212223242526 
Lesson Enactment 


IQA Score Level 


46 


